Peer Collaborative Learning for Online Knowledge Distillation

Traditional knowledge distillation uses a two-stage training strategy to transfer from high-capacity teacher model compact student model, which relies heavily on the pre-trained teacher. Recent online alleviates this limitation by collaborative learning, mutual learning and ensembling, following one-stage end-to-end fashion. However, fail construct an teacher, whilst ensembling ignores collaboration among branches its logit summation impedes further optimisation of ensemble In work, we propose novel Peer Collaborative Learning method for distillation, integrates network into unified framework. Specifically, given target network, multi-branch training, in each branch is called peer. We perform random augmentation multiple times inputs peers assemble feature representations outputted with additional classifier as peer This helps peers, turn optimises Meanwhile, employ temporal mean collaboratively learn richer facilitates optimise more stable better generalisation. Extensive experiments CIFAR-10, CIFAR-100 ImageNet show that proposed significantly improves generalisation various backbone networks outperforms state-of-the-art methods.